Generating Realtime Motion Plans from Attribute-Based Natural Language Instructions Using Dynamic Constraint Mapping

We present an algorithm for combining natural language processing (NLP) and realtime robot motion planning to automatically generate safe robot movements. We present a novel method to map the complex natural language commands into appropriate cost function and constraint parameters for optimizationbased motion planning. Given NLP commands, we generate a factor graph named Dynamic Grounding Graph (DGG). The coefficients of this factor graph are learned based on conditional random fields and used to dynamically generate the constraints for motion planning. We directly map the cost function to the parameters of the motion planner to generate collision-free and smooth paths in complex scenes with moving obstacles. We highlight the performance of our approach in a simulated environment as well as a human interacting with a 7-DOF Fetch robot with complex NLP commands.